The Annals of Applied Probability 

2004, Vol. 14, No. 4, 2038-2056 

DOI: 10.1214/105051604000000873 

© Institute of Mathematical Statistics, 2004 



STABILITY OF NONLINEAR FILTERS IN NONMIXING CASE 

By Pavel Chigansky and Robert Liptser 

University of Tel Aviv 

The nonlinear filtering equation is said to be stable if it "forgets" 
the initial condition. It is known that the filter might be unstable 
even if the signal is an ergodic Markov chain. In general, the filtering 
stability requires stronger signal ergodicity provided by the, so called, 
mixing condition. The latter is formulated in terms of the transition 
probability density of the signal. The most restrictive requirement of 
the mixing condition is the uniform positiveness of this density. We 
show that it might be relaxed regardless of an observation process 
structure. 

1. Introduction and the main result. This paper addresses the stability 
problem of the nonlinear filtering equation with respect to its initial condi- 
tion. We consider a homogeneous ergodic Markov chain {Xn)n>o with values 
in S C M'^ regarded as a signal to be filtered from observation of (i^)n>i) 
Yn G MP. 

Denote M{dx), K{x;dy) and N{dx) the invariant measure, the transition 
probability kernel and the distribution of Xq, respectively, and describe the 
dependence of observation and signal processes via the conditional distribu- 
tion T{x;dy): for any Borel measurable set A from W and -'^[cn]! ^[i,n] the 
cr-algebras generated by {Xq ,...,Xn}, {Yi,...,Yn} 



a.s. 



(1.1) P{Yn G A\X[o,n] V l[l,n-l]) = P{Yn G A\Xn) = r(X„, dy) 

An example of such type dependence is 

^{Xn ) Cn ) > 

with a bounded measurable function h and i.i.d. sequence of random vari- 
ables (Cn)n>i independent of {Xn)n>o- 
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We assume that there exist reference measures ip{dy) on S and 0{dy) on 
W such that 

K{x-dy) = X{x,y)'il;{dy), 

N{dx) = u{x)il^{dx), 

T{x-dy) = -f{x,y)B{dy), 

where the densities A(x, y) and 7(0;, y) are measurable nonnegative functions. 
Then, obviously, M{dx) = 'm{x)il){dx) and P{Xn. € (ix|Y[i „]) has (a.s.) a 
density vr^(a;) with respect to ip: for any Borel set A from S, 



p(x„,eA|y[i,„])= / <(x)v^(dx). 

J A 



It is well known that iTnix), n > 1 solves the, so called, filtering equation 
(recursive Bayes' formula) 

(1.2) 7r„(a;) - 



k k 7(^i, 5^n) A(w, nX„i {v)il}{dv)i^{du) 

subject to vtq (x) = v{x). 

If the density ^'(x) is not completely known, in practice the filtering equa- 
tion is initialized by another density f3{x) corresponding to some probability 
distribution B{dx) on S with respect to ijj. Though this substitution seems 
natural, the fraction in the right-hand side of (1.2) may not be well defined. 
Nevertheless, '\i N -^B [henceforth, N <^B is assumed and ^{x) =: ^(x)], 
the recurrent equation (1.2) subject to [3 makes sense tjj-a,.s. 

If V is replaced by j3, we get "wrong" filtering density ^^^{x) defined as 
follows: TiQ'^{x) = (3{x) and 

av( N lix^Yn) kK^^^)'^n-i{z)'4^{dz) 



k k 7(«> yn)Kv. uy^-i {v)ip{dv)tp{du) 

Since vr^^'^ differs from vr^, the use of 7r^'^(/) := /g /(x)7r^'^(x)V'(dx) as an 
estimate for /(X„), with bounded / or Ef'^{Xn) < 00, does not guarantee 
the mean square optimality, that is, 

E{f{X^) - <(/))2 < i?(/(X„) - vrf (/))2 

even as n 00. 

The filtering equation, or filter, is said to be stable, if for any bounded 
measurable function /, 

(1.3) Jim £;!<(/) -vrf^(/)| = 0, 

so that, from the practical point of view, the use of vr^'^ for stable filters 
makes sense. 



STABILITY OF NONLINEAR FILTERS IN NONMIXING CASE 



3 



Assume that (3 is chosen and consider a new pair of processes {X^,YJ^) 
characterized by the same kernels K and T and (3 as the distribution density 
of X^. It can be shown (similar to Proposition 2.1, [2]) that (1.3) holds 
provided that 

(1-4) ^(^(^o^)|^[f,oo)) =l}n.^E(^^iX^)\x^,Y^Q a.s. 

So, (1.3) follows from >^[f^oo) = nn>o{^[f,oo) ^ -^^(^,00)}' ^'^^^^ ^y the Markov 
property and the martingales convergence theorem. 



}^^E[-{X^)X^,Yl^^J=Ei-iX^] 



n>0 



Unfortunately, in general, yj^^^-, £ n„>o{^[i,oo) ^ ^[n,oo)} ^^^^ though the 
ergodic property of the Markov chain implies r\n>o-^[n 00) ~ i^'^) 

In several publications Yjf^^^ = nn>o{'^[f 00) "^^[I.oo)} implicitly de- 
clared and then (1.3) is derived (for more details, see [2] and citations 
therein). Somewhat contradicting the intuition, the filter does not auto- 
matically inherit (1.3) from the ergodic property of signal. Corresponding 
counterexamples can be found in [2, 7, 8] (see also Example 5.1). 

The validity of (1.3) sometimes crucially depends on the structure of 
7(x,y). For instance, in order to have (1.3), 7(x,y) is required to be com- 
pactly supported (see [4] ) , or to have tails with certain decay rate depending 
on the signal model (see [1, 5, 9, 10, 16]). 

In view of the aforementioned facts, it is interesting to determine the 
conditions providing (1.3) regardless of 7(rc,y). One of them is the mixing 
condition introduced by Atar and Zeitouni [1] and Del Moral and Guion- 
net [6]: 

(1.5) 0<A*<A(a;,y)<A* 
which provides 

(1.6) limsup-loglK -vr^^'ll < a.s.. 



where 



f -<ll= / kf(x)-<(x)|V^(dx) 



is the total variation norm. Clearly (1.6) implies (1.3). 

If after all the condition A* < 00 is reasonable, the condition A* > is 
quite restrictive in many applications. 
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In this paper we replace A* by 

(1.7) X.= [essini X(.,y)mi.Md.) 

(hereafter, essinf and ess sup are taken with respect to V')- 

Theorem 1.1. Assume < Ao and A* < oo. Then, for any ^{x) and 
any f3{x) > > 0, the relation (1.4) holds and 

(1.8) hmsup-logllvrf'' < a.s. 

n n A 

Remark 1.1. In contrast to X{x,y) > A^, > 0, the assumptions of Theo- 
rem 1.1 admit X{x,y) to be zero on some region of the state space. 

Example 1.1. Let X„ be the Markov chain with values in S = {1, . . . , d} 
and A be its transition probability matrix with entries 

Xij = P{Xn= j\Xn~i = i). 

Here Xij is the transition probability density with respect to the Dirac mea- 
sure ip supported at points j = 1, . . . ,d. 

Assume A* = miuj^ Xij = 0, but there is an integer r > 1 such that all 
entries of A'' are positive. Then the Markov chain is ergodic (see, e.g., [14]) 
and its invariant measure possesses density m with respect to ip with positive 
atoms m{i), i = 1, . . . ,d. Then 

d 

Ao = y^min Xijm{i) > 0, 
i=i ^ 

if at least one row of A has positive entries. 

Remark 1.2. The relation m{y) = JgX{x,y)m{x)'ilj{dx) implies 

(1.9) Ao<m(x)<A*. 

Remark 1.3. The requirement /? > /3* > is not restrictive, since the 
choice of /? is flexible. 

Notice also that /3 > /?* implies N <^ B with 



STABILITY OF NONLINEAR FILTERS IN NONMIXING CASE 



5 



Our proof of (1.8) differs from these in [1] and [6]. Before getting into de- 
tails, we describe the main idea of the proof, where the backward conditional 
density 

P{Xoedu\Xn = x,Y[i^^]) 

^"(^'^) = Wn) 

plays a crucial role. 

We show in Section 4.3 that \pn{u,x') — pn{u,x")\ —>0, ?i — > oo, for any 
x' , x" which allows the claim that 

is a Yji oo)-measurable random variable and in turn the validity of (1.4). 

The proof of the second statement of Theorem 1.1 uses an upper bound 
for the rate, in n, of \pn{u,x') — pn{u,x") \ 0. When A* > 0, this rate is 
A* /A*, while for Ao > the rate is random (Lemma 3.2) and controlled by 
7r„(x). The upper bound in (1.8) follows from the law of large numbers for 
the conditional expectations (Theorem 2.2), which is derived with the help 
of geometric ergodicity (Theorem 2.1) being valid under the assumptions of 
Theorem 1.1. 

2. Geometric ergodicity and law of large numbers for conditional expec- 
tations. 

2.1. Geometric ergodicity. Let X^^\u,x) be the n-steps transition prob- 
ability density with respect to ip. 

The Markov chain is geometrically ergodic, if there exist constants C > 
and < r < 1 such that for any u, 

\X'^'''\u,x)-m{x)\i;{dx)<Cr'', n>l. 

Theorem 2.1. Assume < Ao and A* < oo. Then the geometric ergod- 
icity holds with 

C = —— -— and r = l — —. 

Ao(A*-Ao) A* 

Proof. Consider the stationary Markov chain X„, that is, when Xq has 
distribution density m{x) with respect to ^. 

By the Bayes formula, one verifies that P{Xq G du\Xn = x) has a density 
qn{u,x) with respect to ^jJ 



(2.1) 
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and 

_ /§ A(x^ x)qn-i{u, x')m{x')'ip{dx') 

(^Z.ZJ Qni'U, X) — 

■m[x) 

Let us show now that 

7n{u) [A*-"-* (ti, x) — m(x)] 

(2.3) 

[(;n(n,x) — qn{u,x')]m{x)m{x')il){dx'). 



For arbitrary bounded and measurable functions g and /, write 
Eg{Xo)f{Xn) - Eg{Xo)Ef{Xn) 

(2.4) 

g{u) f {x)[X^"'\u, x) — Tn{x)]m{u)ip{du)'ip{dx) 



and 

Eg{Xo)f{Xn) = / g{u)f{x)qn{u,x)m{x)ij{du)ij{dx) 

(2.5) 

g{u)f{x)qn{u, x)m{x)m{x')ip{du)%l){dx)'il){dx'). 



Obviously, (2.5) provides 

Eg{X^)f{Xn) - Eg{Xo)Ef{Xn) 

(2.6) =/ / I 9{u)f{x)[qn{u,x) - qn{u,x')] 

X m(x)m{x')ip{du)ip{dx)'ip{dx'). 

So, by arbitrariness of (7 and /, (2.6) and (2.4) imply (2.3). 
Owing to (1.9), the relation (2.3) provides 



\\^''\u,x)-m{x)\ 

\qn{u,x) — qn{u,x')\m{x)'m{x')ip{dx'). 



(2.7) ^ 

m(n) J§ ' 

Introduce g„(n) = esssup^gg g„(u, r), q^{u) = essinfrg§g„,(M,r) and set 

An(^^) = qniu)-Qju). 

By (2.7), 

(2.8) / |A(")(n,x) - m(x)|^(dx) < ^A„(n). 
Js m[u) 
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We show now that 



(2.9) A„(n)<m(n)^(^l-^) , n > 1. 



For n = 1, this estimate is obvious, whereas qi{u,x) < m{u)X* / X<^, that is, 
Qi{u) < m{u)X* / \o, and Ai(u) <qi{u). In order to estabhsh this estimate 
for n>2, we derive at first a recurrent inequahty 

(2.10) A„(n)< (^l-^^A„_i(n). 

For any v',v", we have 

Is rn{x')X{x', v')qn_i{u)^{dx') 



Qn-iiu) 
Q 1 (^) 

-In— 1^ ' 



m{v') ' 
/§ m(x')A(3;', ';;")9„_i(^^)V'('^a;') 



m(f") 
and, by (2.2), 

qniu,v') - qn{u,v") 

_ Is A(a;^ v')qn-iiu, x')m{x')tp{dx') 
m{v') 

_ Is A(x^ v")qn-i{u, x')'m{x')%l){dx') 
m{v") 

Then 

qn{u,v') - qn{u,v") 

= Qn-iiu)-q^_^iu) 

_ /gm(xOA(x^^O[g■„_l(n) - qn-i{u,x')]'il){dx') 
m{v') 

Is "^(x')A(x^ v") [qn-i{u, x') - q^_^{u)]ij{dx') 
m[v") 

This equahty, with (0/0 = 0.5 is understood here) 

^ f^^y 1n{u,x')-qju) 1 a (ux')- ^n{n) - qn{u,x') 

an{u,x ) .— - ana i — an{u,x).— - ^— 
A„(ti) A„(m) 

is transformed to 

qn{u,v') - qniu,v") 

= An-i{u)(l- f m{x'){^^4r[l-an-i{u,x')] 
\ Js I m(f') 
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H ^-T— On-l [u,x)\'\\){dx)\. 

m{v") J / 



Owing to < ) < 1 and m{x) < A*, we get 

, , fAfx'jf') , A(x',-u") /n1 ,/, /\ 

?n(x TTT- 1 - an-i{u,x)\ H — — q;,„_i(m, x ) }ip{dx ) 

{ m[v') m[v") J 

> / '^[Kx',v')^\{x'ymdx') 

If A 

> — / m(x')essinf A(a;',r)^/;((ix') = -r^- 

Thus, qn{u,v')-qn{u,v") < (1 - A^/A*) A„_i(u), and (2.10) holds true by 
arbitrariness of v' and v" . 

The iteration of (2.10) and the estimate Ai(ii) < m{u)\* / give (2.9). 
So, by (2.8) the result holds. □ 

2.2. The law of large numbers. The result of this section is valid for any 
distribution of X^, obeying density with respect to '(/'• 

Theorem 2.2. Assume A^ > and X* < oo. Then, for any hounded mea- 
surable function f , 



1 " f 
lim -V7rfc_i(/)= / f {x)m{x)'4){dx) 



a.s. 



Proof. Set f°{x) = f{x) — J^f{z)m{z)ijj{dz) and notice that the desired 
statement is valid if 

1 

(2.11) lim -5]^fc_i(r) = a.s. 

k=l 

Let us introduce the Poisson equation 

9{x) = f°{x)+ / g{y)X{x,y)'tp{dy), 

whose solution 

oo „ 

g{x)=f^x) + Y^ / r{y)X^^Hx,y)i;{dy) 

is well defined and bounded by virtue of the geometric ergodicity (see The- 
orem 2.1). 
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Set = 7r„,((7) —TTn-iig) +7i"n-i(/°) and notice that ^„'s forms a bounded 
martingale difference. The boundedness is obvious; E{(^n\Y[i,n-i]) = 0) a.s., 
n > 1, is vahd by the following Poisson equation: 

E{U\Yii,n-i]) = E{g{Xn) - g{Xn-i) + nXn-i)\Y^i,n-i]) 

= E(^J^g{z)XiXn-i,z)i;{dz) - giXn-i) + nXn-i)\Y^i,n^^^ 

= 0. 

So, with the martingale M„ = J2k=i ^k, we have 

n 
k=l 

Consequently, 

1 " 11 

- V 7rfc„i(/°) = -M„ (vr„(5) - 7ro(5')). 

n n n 

fc=i 

With bounded g, iTn{g) can be chosen bounded too. Hence, 

n'^^iT^nig) -T^oig)) 0. 



n— »oo 



Also Mfi/n 0, a.s. n — > oo, since the increments of martingale M„ are 
bounded (see Theorem 4, Chapter VII, Section 5 in [17]). 
Thus, (2.11) holds. □ 

3. The backward distribution P{Xo G du\Xn = x, Yji^n])- The results 
of this section are valid for any distribution of Xq obeying density with 
respect to ip, say bounded below by a constant > 0. 

3.1. Recurrent equation for the density. We show that the backward dis- 
tribution possesses density pn{u,x) with respect to ip. 

Lemma 3.1. Under the assumptions of Theorem 1.1 pn{u,x) exists and 
is defined as 

_ X{u;x)^u) 
P'^''''''>- J^X{v;xnv)^{dv)' 

(3.1) 

/ N JsX{x';x)pn-i{u;x')Trn-iix')i!{dx') 

Pn[u; x) = r-TT—i — ^ / /^ ,M i\ ' n>2. 

jgX{x';x)Trn-i[x')ip{dx') 

Proof. For n = 1, the formula is obvious. 

By (1.1), E{h{Xo)\Xn,Y[,^n]) = EihiXo)\Xn,Y[i,n~i]) for any bounded 
measurable h and n > 2. We apply the induction method. With bounded 
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and measurable functions g{x),f{x), x G S, and H{yi, . . . ,yn-i), Ui & 
write 

Eg{Xo)H{Yi,...,Y^_i)f{Xrr) 

= Eg{Xo)H{Yu...,Yn^i) [ /(x)A(X„_i, x)V'(dx) 

(3.2) 



X Trn-iix')ip{dx)ijj{dx')ip{du) . 

On the other hand, 

Eg{Xo)H{Yi,...,Yn-i)f{Xn) 

= EH{Yi,...,Yn-i)f{Xn) J g{u)dP{XoGdu\Xn,Y^^^^^^) 

(3.3) / 

= EH{Y,,...,Yn-i) / / /(x)g(n)(iP(XoG(i^x|X„=x,l^i,„_i]) 
JsJsJs 

X X{v,x)-Kn~l{v)llj{dx)7p{dv). 

By arbitrariness of /, we derive from (3.2) and (3.3) that (^p x ■0)-a.s. 
dP{Xoedu\Xn,Yiin]) / A(w;x)7r„_i(f)V'((i?;) 



(3.4) 

= ip{du) I X{x',x)pn-i{u;x')7rn-i{x')'il^{dx') 



and the proof is complete. □ 

3.2. Upper bound for \pn{u,v') — pn{u,v")\. Set 
(3.5) p„(n) = esssup/9„(n,x) and p (u) = essinf /9„(n,x) 

and introduce 5n{u) = 'Pn{u) — P^{u)- 

Lemma 3.2. Under the assumptions of Theorem 1.1, 

(A*)^ f 1 " /■ 1 

Sniu) < '0{u)exp{-—Yl / vrfc„i(x')essinf A(x',r)V'((ix') >, n > 2. 
^*A, ( A*^t^2^ r J 

Proof. Obviously, 

(A*)^^(u) (A*)^ 
^'^""'^^ - ^9./sA(^;,x)m(T;)V(dt;) " ^^Ao^""^ 
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and, whereas 6i{u) < pi(u,x), the first statement is vaUd. Further, by (3.1) 
we have 



(3.6) 

Using the identities 

Pn-i{u) 



Pn{u,v') - pniu,v") 

_ J nn-i{x')X{x' ,v')pn-.i{u,x')ip{dx') 

J ■Kn-liv)X{v,v')'lp{dv) 

_ J Trn-i{x')X{x', v")pn-iiu, x')i{;{dx') 

J TTn^l{v)X{v,v")tp{dv) 

J TTn-l{x')X{x' ,v')pn-l{u)'ll){dx') 
j ■Kn-l{v)X{v,v')'4){dv) 

j ■Kn-l{x')X{x\v')p^_^{u)'4j{dx') 



J iTn-i{v)X{v,v')i(;{dv) 



i-n— 1 ^ 

rewrite (3.6) as 

Pn{u,v') - pn{u,v") 

= Pn~liu) - P^^^iu) 

_ / 7Tn-l{x')X{x', V') [p„„i (U) - pn-l{u, x')]^{dx') 
J 'Kn~l{v)X{v,v')ll){dv) 

J TTn-l{x')X{x', v") [pn-1 {u, x') - p^_^{u)]lp{dx') 
J TTn-l{v)X{v,v")lp{dv) 

Now, following the proof of Theorem 2.1, introduce 

a..(.,xO:^ ""<"-f,7""'' and 1 - ...(u.x') := ^"^ -^f'^'' 

dn[U) dn{u) 

and rewrite (3.7) into 

Pn{u,v') - pn{u,v") 

/7r„„i(x')A(x',i;')[l - an-i{u,x')]'ip{dx') 



■ 5n_i(n) ^1 



J Trn-i{v)X{v,v')'4'{dv) 

J TTn-l{x')X{x', v")an-.l{u, x')'ip{dx') 



j ■Kn-l{v)X{v,v")lp{dv) 

Since < a„(n,x') < 1, we have 

/ ■Kn-l{x')X{x' ,v')[l - an-l{u,x')\ll){dx') 
^ ■Kn-l{v)X{v,v')'li){dv) 
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_^ / TTn-lix')X{x',v")an-liu,x')'lp{dx') 
J -Kn-l{v)X{v,v")lp{dv) 

> j7T^Mx')^^f^[l-an-l{u,x'M{dx') 
+ J T^n~l{x') ^^^^^ ^ «n-l {U, x')^{dx') 

>^J 7Tn-iix')[X{x',v')AX{x',v")]i;{dx') 

> — / 7r„_i(x') essinf Afx', r)'0(dx'), 
A* J res 

where the last inequahty is vahd {ip x ip)-a.s. Hence, 

Pn{u,v') - pniu,v") 

<6n-i{u)(^l - ^ J 7rn-i{x')essinf X{x',r)if;{dx' 
and, by arbitrariness of v' and v" , 

(3.8) < (5„_i(n)^l - y"7r„_i(x')ess^inf A(x',r)V'(dx')^. 

The iteration of (3.8) and log(l — x) < —x, < x < 1 provides the second 
statement. □ 

4. Proof of Theorem 1.1. 

4.1. Notation. Let {X^^Y") and {Xf^,Yf^) be copies of {X,Y) when the 
distribution density of Xq is or /3 and Q'^ and be probabihty distri- 
butions of (X^,y^) and {X'^,YP). Also let and be the distributions 
of Y'^ and Y^ and and be the restrictions of Q'^ and on the 
(T-algebras „] and „]• Without loss of generality, we may assume that 
both copies are defined on the same probability space. 

Set 7r(^(/) = £'(/(X^)|yj^^j) and let vr(^(2;) be the corresponding filtering 
density. 

4.2. Absolute continuity <C Q^. Since both Markov chains {X'',Y'') 
and {X^, Y^) have the same transition probability kernel, by the assumption 
N <^ B we have 



Q''«Q' with^(x^y/^) = ^(Xo^), 
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dQ 



dQn, 



Yi 



Y' 



/3 



[1,00) j ' ^ [1,00) 



V yL, 



dQ: 



Y 



l,n 



n > 1. 



4.3. Proof of (1.4). It suffices to show that 



n {^[1,00) v^[i,oo)} 



n>0 



is a Y,T s -measurable random variable. Indeed, then we have a.s. 



n>0 / 



n {^[1,00) V ^[^,00)} 



n>0 



Y 



[1,00) 



/3 



[1,00) I- 



In order to verify the required Yj^^^-measurability, notice first that by the 
reverse martingale convergence theorem, 



n {y.f , vxf J I = lim e( ^(X 

I I L [1,00) [n,oo)J I „_,oo \ ' 

n>0 / 



[1,00) ['^,00) 



and by the Markov property, 



e(-{x^)y^ ,yx^ ,]=e(-{x^)y^ ,vx^^ 

yg\ 0/ [1,00) [n,oo) J OJ [l,n] n 



§/3 



[u)pn{u,X^)lp{du). 



So by the aforementioned martingale convergence theorem, 



lim / —iu)pn{u,x!t)il){du) exists 



a.s. 



On the other hand, since with /0„(ti) and p (n) defined in (3.5) we have 



s/3 



u)p{u)'ip{du)< / ^{u)pn{u,Xf^)ip{du) < / ^{u)p^{u)'4){du) 



14 P. CHIGANSKY AND R. LIPTSER 

and, by Lemma 3.2 with = /3(x), for n > 2, 



< 



(A 



*^2 



/3* Ao 



expj -^^^7rf_-^(x')ess^inf A(x',r)i/;(dx') i. 



By Theorem 2.2, hm^^oo n'^ Y2=2 1 '^k-ii^') essmf,. X{x', r)i;{dx') = (> 0), 
a.s. Consequently, hm„^oo /§[Pn('") ~ = Oi a-s- and, 
therefore. 



n^{<»,v<„,„,})=„to/i(")?'» 



{u)ip{du). 



The desired statement holds since Jgp^{u)v{u)/ f3{u)ip{du) is a ^[1,00)- 
measurable random variable for any n. 

4.4. Proof of (1.8). For fixed n, let (p'^{y[i^n]) and </''^(2/[i,n]) be mea- 
surable functions of the arguments yi, . . . yi gMP, such that for fixed 
bounded and measurable function /, 

r{Y[l^])=<{f) and ^^{Yl^^)=n^Jf). 

Recall that N^B provides Q^-^Qf^. So we may define n^^if) = /(^[i,™]; 
Since <C Q'^ is not assumed, we set 

<^{f) = r{Yl^^)l(^{Yl^^)>0). 

Lemma 4.1. Under the assumptions of Theorem 1.1, for n>2, 
dQ\ 



(4.1) 



_^:(y^)||<^-7rf| 



< 



(A 



*\2 



■ exp 



J ^k-iix')essmiX{x', 



r)il){dx') >. 



Proof. We show first that 
dQr. 



^(y^)(vrf(x)-7r^^(x)) 



(4.2) 



7r^(x) 



§/3 



{u)pn{u,x)'lp{du) 



{u)pn{u, x')Tr!^{x')\lj{dx')\lj{du) 
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Let g and / be bounded measurable functions of arguments yi,. ■ ■ ,yn, Vi £ 
and X e S. Then, 



E 



^[l,n] 



[l,n] 



-E-{X^)giYl^^)K^{f)-f{X^^,)] 

^^5(>^[l,n])K(/)-/(^n)] 





provides 
(4.3) 



(IQl 



(y^K^(/) = i?(^(Xo^)/(xf) 



/3 



Further, by (4.3), 



dQn 



= £(^(xi')/(x£)|y(f,„,) - E[!^(xt,\Yl,;)E(f(x^,)\Yl„ 
= E(/(xf)[E{^(x„'')|xf,v^t.,}-£{|(x„^)|y^f,„,}]|v^t 

and so, due to arbitrariness of /, (4.2) holds true. 

The right-hand side of (4.2) is evaluated from above as follows: 



-{u)pniu,x)llj{du) 

' / ^(.u)pn{u,x')7r^{x')i;{dx')i;{du) 
sJsp 



vrf(x) 



[u)[pn{u,x) - Pn{u,x')]'K'l{x')il{du)lp{dx') 



<'^n(.x) / -{u)6n{u)lp{du) 

Js p 

and the desired result follows by Lemma 3.2 with •& = [3. □ 
By (4.1), 



-log^(l^^)lkf-vrf| 



1 , (A*)2 
<-log^ 



1 " /■ 

— / vrf 1 (x') essinf A(x' r)'i/'(dx') 
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and, since by Theorem 2.2, 

1 ^ 



i f 

lim — / vrf , (x') essinf X(x' ,r)'ih(dx') 
= /m(xOessmfA(x',r)^(dx') = A., 

we get 

~" A 



p(limsupilog|^(y'3)|Kf -vrfllU 
\ n-^oo n idOZ. J 



1. 



■dQ\ 

On the other hand, owing to Q <Q'^, 

Then, the statement of Theorem 1.1 is vahd, provided that 

dQt 

Notice that 



(4.4) hm -log-^(yn =0. 



dQi ■ \dQ' 



Y' 



that is, ^(y^) is a nonnegative uniformly integrable martingale with the 
limit point ^^(y^) such that < ^^(1"^) < oo, a.s. Again taking into 
account <^ , we conclude that 

p( lim ^(y-) = ^(y- 
V™dQj dQ^ ■ 



with 



^(o<^(y^)<oo) = i. 



Moreover, on account of 



\dQ'^ y \dQ^ J dQ^ 

we have that P(0 < ^{Y") < oo) = 1. 

dQ 

Thus, (4.4) is valid. 
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5. Discussion and one counterexample. In this paper the exponential 
stability of the filtering equation is verified for a class of ergodic Markov 
signals whose transition probability density may vanish on a part of the 
state space. 

To the best of our knowledge, the assumption of Theorem 1.1 is the weak- 
est currently known one, when the filter is stable regardless of the observa- 
tions. 

The obtained result, as well as the result based on the mixing conditions, is 
ideally compatible with compact space §. Except for several known examples 
(see, e.g.. Example 1 in [6]) with S = M, these conditions are not applicable 
in many practical filtering schemes with noncompact S. Particularly, the 
Kalman-Bucy filter is out of their scope, though its stability is well known 
(see [3, 7, 12, 13, 15] and [11], Sections 14.6 and 16.2). 

Since both types of conditions imply the geometric ergodicity of the signal, 
it may seem that it provides the filtering stability. Example 5.1 demonstrates 
that this is false in general. 

Example 5.1. Following [8], we consider Markov chain X^, with values 
in S = {1,2,3,4} and the transition probability matrix 



Owing to A* = and = 0, neither mixing nor our conditions are not 
applicable. On the other hand, since 1 is the simple eigenvalue, the geometric 
ergodicity holds for the unique invariant distribution (0.25; 0.25; 0.25; 0.25). 
Suppose that 



In the context of our setting, the measure tjj is defined in an obvious 

way (see Example 1.1) and = (z^i; 1^2; t's; 2^4) and P = {Pi; P2', Ps', f^i) are 
corresponding densities with respect to ip. 

In [8], Kaijser notes that the vector process 7r„ with the entries '7r„(i) = 



A 



/0.5 0.5 \ 

0.5 0.5 

0.5 0.5 

V0.5 0.5/ 



y: = i{x;^ = i) + i{x!^ = 3) 



n > 1. 



P{Xn = i\Y[i^n]), i 



= 1 



2, 3, 4 is not ergodic. We show also that 



4 



TT! 



n 



n 



.V 



^|7rf(A:)-<(A;)| 



fc=i 




rV 
n 1 
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<(2) = [<.i(2)+<_i(i)](i-y„^), 

<(3) = [<-l(3)+<_l(2)]y„^ 
<(4) = [<-i(4)+<_i(3)](l-y„^) 
subject to T^oik) = Uk, A: = 1, ... ,4. Similarly, 

^f(l) = [v^f::l(l)+v^f::l(4)]y„^ 

^f(2) = [vrf::i(2)+vrf::,(l)](l-y„^), 

^f(3) = K^-l(3) + v^f^l(2)]y„^ 

^f(4) = kf^i(4) + 7rf^i(3)](l-y,^) 

subject to ttq^ {k) = /c = 1, . . . , 4. 
By virtue of the obvious identities, 

<(A;)y„^ = <(fc) and vrf (A;)y„- = ^f'^(fc), fc = l,3, 

<(A;)(l-y„-) = <(fc) and Trf - y„-) = (A:), fc = 2,4, 

we find 

|<(1) -vrf (1)1 = -7rf^i(l)+<_i(4) -vrf::i(4)|y„^ 

= K_i(l)-7r^^i(l)|y„%y„'^ 

+ K-i(4)-vrf^:i(4)|(i-y„%)y„^ 

Similarly, 

|<(2) - (2)1 = |<_i(2) - 7rf::,(2)|(l - y,r_i)(l - Y^) 

+ K_i(i)-7rf^i(i)]y„%(i-y„^), 

i<(3) -^f (3)1 = K_,{2,)-7:tmy:-iy: 
(5.1) + i<_i(2) - ^tiim^ - yn^x)K^ 

|<,(4) - (4)1 = |<„i(4) - 7rf::i(4)|(l - y„^_i)(l - y„-) 

+ K-i(3)-vrf::i(3)]y„%(i-y„'^). 

Now, (5.1) provides ||7r^ — t^^\\ = W^^n-i ~ ^n-ill' 'n>2. 
Hence, |K - tt^^W = IK - vrf^'H. Set 

ci = - /3i + - Pa\ + Ws - ^3 + '^2 - P2\, 

C2 = |2^2 - /?2 + i^l - + |l^4 - (34 + 1^3- Psl 
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and assume that ci A C2 > 0. Since 




c,Y{ + C2{l-Y^n>ci/\C2, 




we have ||7r^ — vr^^'^H > ci A C2- 
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